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ABSTRACT 



This study concerns the problem of inauring ready recall of a large 
body of in-flight job information for Air Force pilots^ The effectiveness 
of a voluntary self- tutoring approach utilizing one type of "game appeal" 
device was examined^ Two matched groups of Air Force pilots were pre- 
tested on their knowledge of instrument flying inf ormation. The device 
was then installed in the crew lounge of one of the groups. No device 
was available to the other group. After a 2-month period both groups 
wore post-tested. Despite the fact that mlnlmaJ, exposure to the machine 
occurred, players improved significantly on the criterion tests while 
non-players did not improve. Factors pertinent to the interpretation of 
the results and implications for further research were discussed* 
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IN^rRODUCTIQN 



One of the more critical aspects of the pilot'a Job ia the 
requirement for accurate and ready recall of a massive body of fliglit 
information^ A considerable amount of factual matter ia taught to 
pilots during their initial flying training and in subsequent tran- 
sition programs throughout their careers. To illustrate the problem, 
a B'l\7 pilot is expected to know all of the information in the B-47 
night Handbook, a 607-page volume. In addition, he must know a great 
many STRATEGIC AUR COMMAND STANDARD OPERATING PROCEDURES verbatim. He 
must know all current instrment procedures and regulations* Moreover, 
much of this information is undergoing continual revision so that 
learning o;!* original subject matter i3 not sufficient. A substantial 
portion of the B-32 pilots' Flight Handbook is devoted to Safety of 
Flight revisions - and the B-32 ia a comparatively new aircraft* The 
effort expended on the part of the pilot to retain a high degree of 
familiarity with his job information is reflected in his flight 
proficiency and accident record. 

Complicating the learning problem associated with the acquisition 
and retention of such a massive body of information are several factors 
related to the mature of the subject matter itself. Much of the 
factual material which the pilot must have at his coninand is of an 
isolated nature; e»g», temperature ranges associated with clear icing, 
or required mlniimims associated with varying flight conditions • 
Con3equej;i1:J.y t uiost of the miaterial does not lend itself to organization 
€tround a ceritral concept permitting greater rapidity in the original 
learning situati.on. Retention of isolated information is correspondingly 
difficult. 

There is also a requirement for the maintenance of seldom used job 
information in a state of ready recall • la flying, the most obscure 
fact may attain great importance under certain conditions. It may, in 
fact, be the one item of knowledge which makes the difference between a 
successful flight and disaster* It is well known that factual material 
which is seldom used is poorly retained, yet seldom used information 
makes up much that the pilot should know. 

The Air Force realizes the importance of requiring pilots to main-* 
tain a high order of job -knowledge. Mandatory Annual Instrument Flying 
Examinations represent one program intended to insui'e this requirement. 
The average pilot studies the subject-matter once a year and is able to 
pass the objective-type quiz. Between the periods of examination, however, 
most pilots forget a good deal of this detailed information. 
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A final conaideration concornu tho techxiiquea utilized fox' ijnpurtine 
now inforuiation to pilots. Theae teciiniciuea are the aame aa those used 
for reviewing pilots' knowled^je of old infonnation. Tho typical approach 
involvea either a aeriea of lectures or tho iaauance of lengthy and 
uninteresting publicationa. Aa much of the Uiaterial is unrelated, little 
used, and often of a highly technical nature, a student motivation pro- 
blem exiflts. Daily study, heme reading, lectures and refresher courses 
are often unpopular with the operational pilot. In addition, most of 
the techniques do not require active student participation! hence, the 
conditions under which learning takes place are leas than ideal • 

APPROACH TO THE PROBLEM 

One possible approach to the training problems juat discussed is the 
use of "self-tutorin^« devices. It seems likely that a practical solution 
to the problem of maintaining job-information in a state of ready recall 
lies with frequent practice, and a self -tutoring device offering aome 
sort of game appeal may provide a painless medium for achieving a high 
rate of exposure to the desired learning materials. Several types of 
devices have been developed, including paper-and-pencil techniques as 
well as machines of varying couipleiity. In the study reported, tne 
effectiveness of one type of self -tutoring device as a technique for 
improving pilot training ana retention of in-flight information has been 
evaluated. 

Description of the Self^Tutorin^ Device 

The device employed was a card machine developed by the U. S. Navy's 
Training Devices Center (see Fig. 1). The device presents multiple-choice 
infonnation items to the player. Scoring is based on elapsed time and 
accuracy of response. A score of 20 points may be attained if the button 
corresponding to the correct alternative is pressed within 5 seconds 
following the appearance of the item. For each additional second utilized 
thereafter* a point is deducted from the 20 points possible. No points 
are awarded a correct response requiring more than 25 seconds , or for an 
incorrect response. Following the subject's response, the correct answer 
to the item is e:qposed by the machine to give the player immediate knowledge 
of results. The machine cumulates the player's scores for 10 items* It is 
important to note that the cards fall into a drawer after each play and may 
be shuffled prior to being rostacked, allowing a randomized presentation. 

Item Selection 

Over 600 instrument flying information items were compiled. An item 
pre-test was conducted using 104 Pilot.«3 assigned to the Flying Support 
Squadrons at Mather Air Force Base. Item-analysis was performed to 
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Figure 1. Self -Tutoring Device Employed in the Study 

determine difficulty level of all items* Several criteria were utilized 
in the rejection of items fron the original item poolj 

1. All items correctly einswered by over 8055 of the Mather item 
pre-test group were rejected, 

2« Items judged obsolete as a result of changes in regulations 
were discarded. 

3. Items characterized as ambiguous or poorly written by the 
Mather item pre»test group were discarded* 

4« Lengthy items « or items with accompanying diagrans which 
would not lend themselves to typing in the restricted space 
available on the card, were discarded* 

The above criteria were used in arriving at the required 3^0 item 

pool* 

VAIXJ TR 39-320 3 
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The atudy waa deait^iod witu two objectivesj tho primary objective 
v<«.y to oxmine tho effectiveness of a self- tutoring device as a technique 
for iiuportin^;^ job information to pilots; the secondary objective was to 
exiuiiino dolayod i*etention of self-tutored information after a reasonably 
lou^; poriod of tiuao had elapsed. » The design called for three equivalent 
tcnit foms« Two foruu (A, B) were used to pre and post-teat; a third 
I'onii (C) was to be administered froui 6 to 12 months after the post-test 
ail the delayed retention measure. The forms were constructed, and items 
corrcnjponuint, to all three forms were prepared for use in the machine by 
a lamination process (see Figure 2)» 

It was deemed desirable to obtain some measure of frequency of 
eocposuro to the machine and its effect upon learning and retention. One 
method of arriving at a measure of frequency of exposure to the machine 
consisted of analysis of questionnaire data* The questionnaire admin- 
istered with the post-test would elicit information with respect to 
"i^emembored niimber of plays" over the 2-month period; however, these 
data woiild be somewhat unreliable due to error in memory on the part 
of the pilots. Consequently, an alternative approach to the problem of 
measuring exposure to the machine was devised. The plem called for a 
diiTerential frequency of item exposure in the machine; that is, some 
iteiiL3 would be repeated in the machine pool while others would not. No 
laminations for the machine were made for one-fourth of the item pool^ 
one lamination was made for each item in the second fourth of the pool, 
two laminations were luade for each item in the third fourth of the pool, 
and three laminations were made for each item in the remaining fourth 
of the pool» This produced four classes of item representing a ratio of 
item exposure in the machine of Ot li 2j 3. The classes are labeled^ 
■non-i^chine items", "once items", "twice items", and "three times items". 
Introduction of the latter three classes of items into the machine 
permitted manipulation of the relative frequency of machine item 
exposure. Play on the machine would result in a differential frequency 
of exposure to tho three classes of machine items* Finally, the effect 
of this differential machine item exposure could be measured by the 
criterion tests employed. 

In designing the experiment careful consideration was given to the 
control of motivational factors. Uader certain conditions, including 
knowledge of their role and importance in an experiment, subjects may 
become hi^:shly motivated to succeed in the experimental situation studied. 



♦ This study was terminated before data could be collected on delayed 
retention; consequently, only the primary objective is considered in 
this paper. 
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Xou art fljlxig on tte XIA^tha airoratt 
heading It the mum tt ttat ZXB tuiMjr 
approach btarl&g. Afttr pitting tht loonl 

istr transmitttr you find jo\x ar* off 

COUTit 

a. You will get an tutoaatio vtm^ofif 

You would rtotlvt no dtflnlta indioat- 
ion on tht oourst dtvlatlon Indicator 
c . Corrtctlont to tht Xocallsar btaa art 
made auay fron tht courtt dtf. Ind* 
d Corrections to tht Xooaliitr btam art 
ntcit into the courtt dtr, Ind* 



Figure 2* Typical Item Laminated for Use in the Machine 



Such a phenomenon has been termed the "Hawthorne" effect after the 
studies carried out by Roethlisberger and Dickson at the Hawthorne 
Plant of the Western Electric Company (4). A "Hawthorne" effect, if 
it were to occur in this study, could introduce a source of spurious 
improvement aa measured by the criterion tests. There was no feasible 
way to prevent the possible existence of a slight Hawthorne effect 
during the post-test, as subjects imnediately recognized the items 
from the pre-test and frcm the machine* Consequently, it was necessary 
to "brief" them on the study at this point in order to obtain their co- 
operation. Increased motivation in the post- test situation could result 
in measured improvements independent of play on the machine. However, 
improvements attributable to this effect would be random with respect 
to the relative frequency of machine item exposure. A significant 
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within-wouj) tayt of dirforontiol lourxiiiv. Uikiutj plnoo tia a iMnction 
or Uiu x^elntivo fxu^iuonoy of moUino item oxpoaura would aubotaatitdly 
reduce the probability of the prt^^^onca of a HawtUorno effect, tuid, In 
fact, oliuduuto the pouuibility of itu buiufc; the lono contributor to 
the iiuprovcuut^uta mouaured* The tmprovmuout would atm fraa inordinate 
interest in tJie niaohiue and in tha learning of iiiachine itema durlnfi the 
oxporiiiiontQl period. Spurioua iu^roveiutint due to a •Hawthorne* effect 
was prevented by withholding: from the aubjecta, knowledge that on 
experiment waa bein^ conducted euid/or that a poot-toat woidd be adiainie- 
torad* Thid will be diacuaaed in detail in a later aection of thia 
ropor t . 

Tha criterion fonas con bo conaidered aa consiatlng of four aoalea, 
each scale consisting of 4^ itonia from each of the four clasaes of items 
described above. Gains on the non-machine itema scale would provide a 
baso line attains t which galna on the tliree machine item scale j might be 
evtuLuated within the experimental tiroup. In evaluating auch a comparisont 
however, the possible influence of the machine on the behavior of the 
experixaental group with respect to instrument job-knowledge "in general* 
would have to be considered* Play on the machine might stimulate 
controversy over items, resultin^j in discussions over coffee and refer- 
enced to official source materials* "Incidental learning" might take 
place under these conditions (3)» Such a phenomenon, if it were to occur ^ 
would be evidenced by a gain on the "non-machine items" scale. 

The effects discussed above, combined with the characteristic 
"practice effect" expecved in a test-retest design, dictated the 
necessity of a control group for the study* The groups were naiaed the 
Device Group and the No Device Group« It is most probable that the No 
Device Group would be subject to the "practice effect" alone. A 
si^^nificant difference between the two groups in gains on the non- 
machine items scale would indicate the presence of either the "Hawthorne" 
or incidental learning effect, or both, in the experiment* It would then 
be necessary to take these effects into account in the interpretation of 
the results* 

The assignment of items to the various scales within the three test 
forms and to the machine is shown in table Ir The forms were composed 
of 80 items common to all three forms and iiO items specific to each form* 
Items were assigned to the scales by a method of matched item difficulty- 
indices* This technique assured equal means and variances with respect 
to item difficulty as determined by the Mather item pre-test* 

Construction of the criterion test forms provided for two types of 
retest situation* From table 1 it may be seen that half of the items in 
each of the four frequency-of-exposure scales were identical from form to 
form; the remaining items were nonidentical form to form but of equivalent 
difficulty level* Greater practice effect could be expected to occur for 
the identical item retest situation than for the equivalent item retest 
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situation. Also, a higher reliability coefficient could be expected to 
obtain for the identical iteiii subscales than for the equivalent item sub- 
scales* IHiree reliability measures were necessary to evaluate the 
criterion tests* Data provided by the No Device Group produced: a test- 
retest reliability coefficient for the 80 identical items (r = •82» n = 36), 
an equivalent form reAability coefficient for the 60 non-identical items 
(r = .71, n = 36), and a measure of form-to-form "equivalent and stability" 
arrived at by simply computing a product-moment correlation between total 
pre- and post-test scores (r =: •84» n - 3^)* These r's were considered to 
be satisfactory* 

Questionnaire 

A questionnaire was developed for administration to the Device Group 
with the post-test. It was believed that attitudinal data» in addition 
to reported behavioral information, mi^t provide additional insight in 
evaluating the effectiveness of the self-tutoring experience examined in 
this study • 

EKPERBISNTAL PRCGEDIRES 



Selection of the Experimental Groups 

Pilots assigned to two Air Refueling Squadrons in the Strategic Air 
Command were utilized in this study • The Device Group was located at 
March Air Force Base, California* The No-Device Gpoup was located at 
Mountain Heme Air Force Base, Idaho* In addition to ascertaining that 
the assignment of pilots to the tvo squadrons was random with respect to 
factors considered relevant to the experimental variables to be examined, 
the squadron missions, flying duties, type of aircraft flown and training 
procedures were determined to be identical • Flying experience and distri- 
bution of grade did not differ significantly for the two squadrons. Sub- 
sequent analysis of the pre- test scores revealed no statistically 
significant difference between the tv^o squadrons in knowledge of the 
items utilized in the study (see ResiJLts). 

Table 2 presents information concerning the N's obtained for the 
study* A loss of subjects occurred due to pennanent change of station, 
flying duty, temporary duty away from the station, leave, or other 
official reasons for absence during either of the testing periods* 
Responses on the post-test questionnaire indicated that I3 subjects in 
the Device Group did not play the machine. ThjLs Device Non-Player Group 
was treated separately from the Device Player Group in the analysis of 
the data* 
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TABLE 2 

SCHEDULE OF SUBJECT PAETICIPATICN 
IN ma TEST ADMmiSTRATIOriS 







^10 






DEVICE 


DEVICE 




SCHEDULE 


GRonp 






Pre-Tested Form A 




27 




Pre-Tested Form B 


21 


28 




Total Ere-Tested 


62 


55 


117 


Post-Tested Form A 


28 


19 


47 


Post-Tested Form B 


20 


iZ 


iiZ 


Total Post-Tested* 


58 


36 


94 



* These figures include pilots present at both test administrations only. 
Twelve pilots not present at the pre-test wer^'e post- tested; however » 
these data were not used in the study. The questionnaire data reflects 
the total Device Group post- test N of 62. 



Procedures 

Pilots assi^i^ed to both experimental groups were pre-tested prior to 
the installation of the self- tutoring device* The equivalent forms (A and B) 
were administered alternately to all subjects. The device was then 
installed j-n the crew lounge of the Device Group for two montus. No device 
was available to the control group. Both groups were then post- tested 
with the equivalent form of the test administered in the pre-test, i«e., 
one-half of the pilots in each group were pre-tested on Form A and post- 
tested on Form B, the other half of each group were pre-tested on Form B 
and post-tested on Form A. The questionnaire was administered to the 
Device Group only. Form C was not administered in this study; however, 
the machine items corresponding to this form comprised one-third of the 
machine pool. 

Conditions Surrounding Voluntary Participation in the Study 

Prior knowledge of any test administration might resxilt in some form 
of subject preparation and it was imperative that both groups be totally 
unaware of their participation in the evaluation of the device. An important 
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aspect of the evaluation of the self- tutoring experience examined in this 
study concerned the nature and frequency of voluntary exposure to the 
device on the peirt of the pilots# Knowledge of an impending post-test 
might influence not only the frequency with which pilots would volur-teer 
to play the machine, but also the nature of the learning situation taking 
place during the experience* Trior to the post- test, a form of "cramming 
on the machine could develop, destroying the objective of the evaluation. 
The briefings delivered at the pre- test administrations were carefully 
prepared so as to prevent the arousal of suspicion of any kind» The 
pilots were told that all 13th Air Force bases would receive machines 
(a true statement) and that the purpose of the test was to provide data 
for item analysis of the large pool of items to be developed for the 
machines* They were told that many of the items were unsuitable, and 
that each should note on the test booklet those items that were poorly 
written, obsolete, contained no correct answer, contained more than one 
correct answer* etc», in addition to answering all items* The fact that 
13th Air Force was actively procuring 65 machines for its I7 bases made 
.the instructions plausible in addition to preventing the arousal of sus- 
picion over the installation of a machine the week following the pre-test 
Wo briefing or incentives to play of any kind, other than those supplied 
by the game appeal of the device itself, were operative during the study* 
All pilots in the Device Group were asked in the post-test questionnaire 
whether they suspected they would have to take a second test, and 100^ of 
the group answered this item "No*" Two individuals in the squadron were 
briefed on the study: the squadron commander and the sergeant assigned 
the responsibility of maintaining the machine and keeping a record of 
the total number of cards restacked daily* 



RESULTS AND DISCUSSION 

The err or- score means and SDs for the experimental groups on the 
criterion tests are shown in table 3* The test of the difference between 
the raw error-score means of the Device Player Group and the No-Derioe 
Group on the pre-test was not significant (t = .971 1 df « #78)* Thus, 
there appears to be no bias in the selection of the experimental groups 
with respect to knowledge of instrument flying information as measured 
by the criterion tests* The difference between the pre-test error-score 
and post-test error-score was derived for each subject in the study* 
The resulting gain-scores, used in all further analysis of the data, 
were independent of initial difference levels on the pre-test* 

TABLE 3 

ERRCR-SCORE MEAllS AND STANDARD DEVIATIONS FOR WE 
EXEERIKENTAL GKOUES ON THE CRITERION TESTS 



Group Pre-Test Post-Test 

N M SD M SD 

Device Players Group 43 60T00 lT743 33 •27 l^Tbl 

Device Non-Player Group 13 38.20 13*10 37*46 13*30 

No Device Group 36 62.80 13-41 62*33 13*20 
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Since this experiment corresponded to complex design of the "mixed" 
type described by Lindquist (2, p, 292), his suggestions for appropriate 
analysis of variance procedures were followed in comparing the gains of 
the Device Player and I^o-Device Groups. Because there were 8 gain-scores 
for each of the 79 subjects (representing each scale frequency for 
identical and equivalent subscales), there is a total of 632 scores in 
the analysis. Table 4 summarizes the design and presents the means and 
SD»s of the subgroups; and table 3 presents the analysis of variance with 
the appropriate error terms. 

The Experimental Groups main effect tested by the Betveen-Sub jects 
error term yielded an F-ratio significant beyond the .01 level of 
confidence (F « 8.64, df « 1 and 77). The results of this test support 
the major hypothesis that an improvement in knowledge of instrument 
flying infonaation would result from the use of a self- tutoring device. 

All other tests of significance were ptirt of the wi thin-sub jects 
analysis^ The following tests were madei Frequency and P x E against 
error], (w). Practice and Effect and P x E against errorg (w), and E x P 
and E X P X E, against error^ (v)« The only significant wi thin-subjects 
cCD5)arison was that between gains attributable to identical item sub- 
scales and gains attributable to equivalent item subscales. Practice 
Effect mean square tested by the within-subject erroTg term was 
significant at the .01 level (E . 10.71, df . 3 and ??)• 

Trend Analysis 

Because frequency of item exposure is a "regular increasing" 
independent variable, a more appropriate analysis of this variable is 
provided by a trend test. The technique utilized > following Lindquist* s 
discussion (2, p, 3Z|2), tests for the presence of a significant wi thin- 
group trend over the extremes of the expected sequence. Using this 
technique, a matched t test of the difference between mean gains 
achieved on the non-machine items scale and mean gains achieved on the 
"three times items" scale by the Device Player Group was significant 
beyond the .01 level (matched t^ - 3.06, df s 35) • Figure 3 presents 
the mean gains on the various exposure frequency scales for three groups. 
The Device Non-Player Group was not included in the analysis of variance 
but is presented in Figure 3 and in the section on questionnaire results 
for comparative purposes • It may be safely inferred from the trend test 
reported, and from inspection of the increasing monotonic sequence 
obtained for the Device Group » that learning took place as a function 
of the relative frequency of machine item exposure. Further, figure 3 
shows that no significant differences in imgprovement on the "non-machine 
items" scale resulted for the experimental groups. This comparison may 
be interpreted as a test for the presence of uncontrolled effects. No 
significant Hawthorne or incidental learning effects were demonstrated 
by this comparison; thus the possibility of any bias in the results due 
to these effects may be dismissed for lack of evidence* 
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Figure 3» Relationship between Item Exposure on 
the Machine and Group Perfonnance 
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;i.uestiormaire Results 



The most important finding frou questionnaire data is depicted by 
a bar diagram (see figure 4). The reported frequency of play on the 




FREQUENCY OF PLAY 

Figure 4» J^equency of Reported Play on the Self -Tutoring Device 



machine should be noted. Total reported play was checked against the 
actual number of items restacked during the study» Although airman and 
non-pilot officers were observed to have played the machine at least as 
often as the pilots assigned to the study, the play "reported" by the 
Device Player Group accounts for about ^5% of the total play recorded. 
From this observation, the infoimation in figure 4 would appear to be 
exaggerated er^l shoiild be discounted considerably. The amount of play 
taking place during the study was small by any standards • The average 
Device Player Group pilot played less than 30 minutes duriDg the entire 
2-month period* 

* 

The investigator did not consider the questionnaire findings to be 
of sufficient importance for detailed examination in this report; however, 
a summary of results is presented in table 6. 
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TABLE 6 

SUM^iARY OF Q^HEoTIOKNAIRE RESULTS * 



Device Device Device Group 

Items (Condensed) Players Non-Players Total 

(n:46) (n::l6) (N=62) 



% % % 



5. "By watching others play».» 
able to read additional 
i terns Yes 

No 

8. "Controversial nature of 
seme iteiBS resulted in:" 

a* arguments with other 
pilots 

b. references to official 
sources 

c. loss of interest in 
machine 

d. other 

10. "Greatest incentive to play:* 

a. geiiibling opportunity 
b# improve self on items 
c* challenge of items them- 
selves 

d# nothing better to do 

11. Would have played more if 
items more interesting^ Yes 

No 

12. " Other pilots attitudes 
toward machine werei* 

a. extremely favorable 

b. very favorable 

c. neutral 

d. negative 

e# felt it a waste of time 



63 31 55 

33 50 37 



.54 19 45 

39 13 32 

7 13 8 

906 



0 0 0 

37 31 35 

50 19 42 

17 13 16 

74 

22 



4 13 6 

48 19 40 

44 25 39 

2 0 1 

0 0 0 
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TABLE 6 (continued) 





Items (Condensed) 


Device 
Players 
(nr46) 
% 


Device 
Non-Players 
(nri6) 

% 


Device Group 
Total 
(N=62) 
% 


14. 


"I personally:" 










a* learned new facts from 

machine 
b» boned up on. . .forgotten 

facts 

c. derived no benefit from 
machine 


24 

67 
20 




— 

— 




"SAG should have a macnine 
in each squadron Yes 

No 


80 
17 


50 

25 


72 
19 


17. 


"I was able to play as often 
as I liked." Yes 

No 


46 


31 

/t4 


42 


♦ Percentages are based on an N of 62 


as data 







from four pilots available for the post-test 
only were included • Percentages do not add 
to 100 due to omissions or responses to more 
than one alternative for some items. 



CONCLUSIOIC 

The self- tutoring experience examined in this study resulted in 
statistically significant gains in job-infoimation for pilots voluntarily 
playing the device* The statistical assurance that success of the self- 
tutoring approach applied to pilot training and retention by them was 
not accidental constitutes em important finding* The fact that improve- 
ments resulting from the application of the self- tutoring technique 
employed in this particular study were not large in no way detracts from 
it, as conditions affecting pilot pairticipation on the device were far 
from ideal* The crew lounge, in addition to being undesirably situated, 
was unpopular with the pilots* JXiany pilots did not visit liiis Icunge 
more than once or twice during the entire experimental period. No external 
incentives of any kind, such as contests or briefings, were employed to 
promote interest in the device* Probably the most severe handicap to a 
fair evaluation of the self- tutoring technique examined involved the 
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specific device employed. The device req.uired almost continual maintenance 
due to malfunctions directly attributable to the ineffectiveness of its 
design. As it was impossible to arrange for full-time monitorship of the 
device, it was out of order for a considerable amount of time, which 
discouraged play* 

The stimulus situation presented to the player by the device, when 
operating, correctly » was considered challenging and appropriate in the 
self- tutoring experience examined^ It should be relatively easy to design 
a device, incorporating the functions and characteristics desired, with 
mechanical and electrical reliability vastly superior to that of the 
device employed.* It is believed that a " troublefree" device of this type, 
in the presence of play-motivating incentives over and above those 
supplied by the game appeal of the device itself would produce worthwhile 
effects in practice. The self-tutoring approach to pilot training in, 
and retention of, in-flight job information appears profitable; the 
q^uestions remaining concern methods of employing the techniq,ue to the 
greatest advantage. 



♦ In the course of the study a highly reliable device was designed for 
construction. This device would employ a l6mm projector in place of 
the laminated card presentation* The major source of malfunction in 
the I^avy device was inability of the equipment to handle the cards. 
They warped, split and were otherwise damaged by the mechanical knife 
used to transport them within the machine. 
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